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Enrolling in college and completing a degree or certificate 

is one of the primary pathways to economic success. In 
2018, median weekly earnings of full-time workers with a 
bachelor’s degree were 57 percent higher than full-time 
workers with a high school diploma only. Despite the 
economic advantages of educational attainment, only 

33 percent of students enrolled in two-year institutions 
complete their programs within three years, and 62 percent 
of students enrolled in four-year institutions complete 

their programs within six years.' Several programs have 
been designed to support college enrollment and retention 
among college students. One example is Bottom Line. Bottom 
Line provides intensive advising for low-income high school 
students, most of whom are the first in their family to go to 
college. The advising is designed to help students apply for 
college and financial aid and select a high-quality, affordable 


institution. For students who attend one of Bottom Line’s 
target colleges which they identified as providing a high- 
quality education at an affordable price, Bottom Line 
continues to provide regular support to students on campus 
for up to six years.? 


This What Works Clearinghouse (WWC) report, part of the 
WWC’s Transition to College topic area, explores the effects 
of Bottom Line on enrolling and progressing in college. 

The WWC identified two studies of Bottom Line, one of 
which meets WWC standards. The evidence presented in 
this report is from one study of the impact of Bottom Line 
on high school and college students—including African- 
American, Hispanic, and Asian students—based in Boston, 
New York City, and Worcester, Massachusetts. 


What Happens When Students Participate in Bottom Line?? 


The evidence indicates that implementing Bottom Line: 
e may increase college enrollment 


e may increase progression in college 


Findings on Bottom Line from the one study that meets 
WWC standards are shown in Table 1. The table reports an 
effectiveness rating and the number of studies and students 
that contributed to the findings. 


Table 1. Summary of findings on Bottom Line from one study that meets WWC Standards 


Average performance 


Evidence meeting WWC standards 


(study findings) (version 4.0) 
Number of Number 
Outcome domain Effectiveness rating Intervention group Comparison group studies of students 
College enrollment Potentially positive effects 81% enrolled 70% enrolled 1 2,422 
Progressing in college Potentially positive effects 80% progressing 71% progressing 1 1,429 


Note: Average performance values (study findings) are generated by one analysis conducted for each outcome, as reported by Barr & Castleman (2017). The college enrollment 
outcome reported in this study represents enrollment in a 4-year college. The progressing in college outcome reported in this study represents being enrolled in college for three 
or more semesters after high school. The effects of Bottom Line are not known for other outcomes within the Transition to College topic area, including middle school academic 
achievement, high school academic achievement, middle school attendance, high school attendance, college readiness, staying in high school, progressing in high school, 
completing high school, college attendance, college academic achievement, college degree attainment, and labor market outcomes. 


BOX 1. HOW THE WWC REVIEWS AND DESCRIBES EVIDENCE 


The WWC evaluates evidence based on the quality and results of reviewed studies. The criteria the WWC uses for evaluating 
evidence are defined in the Procedures and Standards Handbooks and the Review Protocols. The studies summarized in this report 
were reviewed under WWC Standards (version 4.0) and the Transition to College topic area protocol (version 4.0). 


To determine the effectiveness rating, the WWC considers what methods each study used, the direction of the effects, and the 
number of studies that tested the intervention. The higher the effectiveness rating, the more certain the WWC is about the reported 
results and about what will happen if the same intervention is implemented again. The following key explains the relationship between 


effectiveness ratings and the statements used in this report: 


Effectiveness rating 


Rating interpretation 


Description of the evidence 


Positive (or negative) effects 
outcome 


The intervention is likely to change an 


Strong evidence of a positive (or negative) 
effect, with no overriding contrary evidence 


Potentially positive (or negative) effects 


The intervention may change an outcome 


Evidence of a positive (or negative) effect with 
no overriding contrary evidence 


No discernible effects 


The intervention may result in little to no 
change in an outcome 


No affirmative evidence of effects 


Mixed effects 
onan outcome 


How is Bottom Line Implemented? 


The following section provides details of how Bottom Line 
was implemented. This information can help educators 
identify the requirements for implementing Bottom Line 
and determine whether implementing this intervention 
would be feasible in their districts, schools, or colleges. 
Information on Bottom Line presented in this section 
comes from the one study that meets WWC standards (Barr 
& Castleman, 2017) and from correspondence with the 
developer. 


¢ Goal: Bottom Line aims to help students to enroll in high- 
quality, affordable colleges and persist in college to attain 
a degree. 


¢ Target population: Bottom Line is designed for students 
from low-income backgrounds, most of whom are the first 
in their family to go to college. Students of color comprise 
the majority of those served by Bottom Line. The majority 
of students learn about Bottom Line through word of 
mouth referrals, but Bottom Line also actively recruits 
students through recruitment events at local college fairs, 
as well as through schools and local organizations. They 
target school districts in each region and specific schools 
within each district. 


¢ Method of delivery: Bottom Line advisors provide 
individualized guidance to high school students in 
preparing college applications, applying for financial aid, 
searching for scholarships, and selecting colleges that 
align with each student’s goals. Bottom Line advisors at 


The intervention has inconsistent effects 


Evidence includes studies in at least two of 
these categories: studies with positive effects, 
studies with negative effects, or more studies 
with indeterminate effects than with positive or 
negative effects 


target colleges provide the ongoing structure and support 
needed for students to graduate and achieve their career 
goals. 


Frequency and duration of service: High school 
students meet with their Bottom Line advisors for an hour 
every three to four weeks during the college application 
and selection process. College students meet with their 
Bottom Line advisors three to four times per semester. For 
both high school and college students, additional contacts 
are held between meetings through a combination of 
texting, email, and video or phone calls. 


Intervention components: The Bottom Line college 
advising model includes Bottom Line Access for high school 
juniors and seniors, and Bottom Line Success, which entails 
advising for those students who attend a target college. 
See Table 2 for additional details. A new initiative called 
Career Connections was launched in 2018 to strengthen 
Bottom Line’s capacity to build students’ career readiness 
skills and strengthen pathways to the experiences and 
professional connections needed to launch a career. 


Comparison group: In the one study that contributes 
to this intervention report, students in the compari- 


son group did not have access to Bottom Line support, 
but they were able to access existing college advising 
support services. 


N 


Table 2. Components of Bottom Line 


Key component 


Description 


One-on-one college 
application advising 


Financial aid support 


College selection 
guidance 


Transition assistance 


College orientation 
and continued 
support 


Bottom Line advisors begin advising students between the end of their junior year in high school and the beginning of their 
senior year. They have average caseloads of 65 students. Students meet individually with their advisor for about an hour 
every three to four weeks at Bottom Line’s office within their community. Using the LEAD Model (Lists, Essays, Applications, 
Decisions), advisors help students compile lists of potential colleges to consider, write essays, and complete their applications. 


Bottom Line advisors encourage students to apply to a set of target colleges and universities in the area that they have 
identified as providing a high-quality education at an affordable price. In each of the three major cities in which Bottom Line 
is located (Boston, New York City, and Chicago),* Bottom Line has about ten to twenty target institutions they recommend to 
students. 


College affordability is a principal focus for Bottom Line. Advisors help students complete the Free Application for Federal 
Student Aid (FAFSA) and supplementary financial aid forms. They also help students search for additional scholarships and 
make informed decisions about the affordability of each school they are considering. 


Bottom Line advisors help students review college acceptances and financial aid packages, ensuring they have a good 
understanding of the cost of attendance and select a college that is aligned with their goals. 


Approximately 50 percent of students choose to enroll in one of the target institutions and can continue into the Bottom Line 
Success program. In the summer after high school, advisors support students with completing all enrollment requirements, 
including scheduling placement tests, preparing a payment plan, and attending orientation. 


Bottom Line Success advisors are located at each target institution. Advisors have an average caseload of 85 students and 
meet with students about three to four times per semester to provide the following supports connected to their DEAL model: 


¢ Degree — including connections with tutoring, advising, and other resources 


¢ Employability — including identifying students’ desired career, building a strong resume, strengthening their networks, and 
connecting to career-relevant experiences 


¢ Affordability —- ensuring students complete financial aid requirements and have a plan to pay their tuition bills each 
semester 


¢ Life - including adjusting to college life, getting involved with extracurricular activities, and balancing their workload with 
other activities and responsibilities 


Bottom Line continues to provide support to students while they are in college for up to six years. 


What Does Bottom Line Cost? 


This preliminary list of costs is not designed to be exhaus- foundations. The total cost per student receiving intensive 
tive; rather, it provides educators an overview of the major advising through the Bottom Line Access and Bottom Line 
resources needed to implement Bottom Line. The program Success programs is approximately $6,000 over a five- to six- 
costs described in Table 3 are based on the information year period. These costs are paid for by Bottom Line, and the 
available as of August 2020. target colleges provide access to their facilities. Students and 


families incur no costs. 


Bottom Line is a 501(c)(3) nonprofit organization that is 
nearly fully funded by private and corporate donors and 


Table 3. Cost ingredients for Bottom Line 


Cost Ingredients 


Description Source of funding 


Personnel 


Bottom Line is staffed by full-time advisors with college degrees who provide direct services to high Bottom Line 
school and college students. Advisors spend their first year learning the organization's curriculum, 

building relationships with students and staff, serving their caseload, and mastering advisor 

competencies. Returning advisors take on additional responsibilities, such as helping to plan career- 

readiness events or organizing care package assembly events for students. Program administrators 

support the program and conduct recruitment activities. 


Facilities 


Facilities costs include space for administrative staff and for providing services. Bottom Line Access Bottom Line; colleges 
has offices in each of the communities served where high school students meet with their advisors. 

Bottom Line Success has offices in each target college they work with where college students can 

meet with their advisors. 


Equipment and 
materials 


Standard information technology (IT) equipment such as computers, phones, printers, and Bottom Line 
photocopiers for staff is required. 


Direct assistance 


Colleges may waive the application fee for Bottom Line participants. Colleges 


For More Information: 


About Bottom Line 
50 Milk St, 16th Floor 
Boston, MA 02109 
Attn: Steve Colon 


Email: scolon@bottomline.org Web: https://www.bottomline.org/ Phone: (857) 415-4810 


About the cost of the intervention 
Web: https://www.bottomline.org/ 


Research Summary 


The WWC identified two studies that investigated the 
effectiveness of Bottom Line (Figure 1): 


¢ One study meets WWC group design standards without 
reservations 


¢ One study is ineligible for review 


The WWC reviews findings on the intervention’s effects on 
eligible outcome domains from studies that meet standards, 
either with or without reservations. Based on this review, 
the WWC generates an effectiveness rating, which summa- 
rizes how the intervention impacts, or changes, a particular 
outcome domain. The WWC reports additional supplemen- 


Figure 1. Effectiveness ratings for Bottom Line 


| study meets WWC 0 studies meet WWC 


standards with 
reservations 


standards without 
reservations 


Contribute to effectiveness ratings 


tal findings, such as enrolling in a two-year college, on the 
WWC website (https://whatworks.ed.gov). 


These supplemental findings and findings from studies that 
either do not meet WWC standards or are ineligible for 
review do not contribute to the effectiveness ratings. 


The one study of Bottom Line that meets WWC group design 
standards reported findings on college enrollment and 
progressing in college. No other findings in the study meet 
WWC group design standards within any outcome domain 
included in the Transition to College topic area.* Citations 
for the two studies reviewed for this report are listed in the 
References section, which begins on page 8. 
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Bottom Line has potentially positive effects on college enrollment 


The WWC determined that one study that meets WWC group design standards without reservations (Barr & Castleman, 
2017) showed evidence of a positive and statistically significant effect of Bottom Line on college enrollment. 


Bottom Line has potentially positive effects on progressing in college 


The WWC determined that one study that meets WWC group design standards without reservations (Barr & Castleman, 


2017) showed evidence of a positive and statistically significant effect of Bottom Line on progressing in college. 


Main Findings 


Table 4 shows the findings from the one Bottom Line enrollment is potentially positive effects, indicating evidence 
study that meets WWC standards. The table includes of a positive effect with no overriding contrary evidence. 
WWC calculations of the mean difference, effect size, and This finding is based on 2,422 students. The effectiveness 
performance of the intervention group relative to the rating for progressing in college is potentially positive effects, 
comparison group. Based on findings from the one study that indicating evidence of a positive effect with no overriding 
meets WWC standards, the effectiveness rating for college contrary evidence. This finding is based on 1,429 students. 


Table 4. Findings by outcome domain from the study of Bottom Line that meets WWC Standards 


Mean WWC calculations 
Sample Intervention Comparison Mean Effect | Improvement 
Measure (study) Study sample size group group difference size index p-value 
Enrolled in a 4-year Full sample 2,422 80.60 70.30 10.30 0.34 +13 <.01 
college (%) 
a Statistically 

Outcome average for college enrollment (Barr & Castleman, 2017) 0.34 +13 significant 
Continuously Cohort 1 1,429 80.40 70.50 9.90 0.33 +13 <.01 
enrolled for the 3 
semesters following 
high school (%) 

ingi a Statistically 
Outcome average for progressing in college (Barr & Castleman, 2017) 0.33 +13 significant 


Notes: For mean difference and effect size values reported in the table, a positive number favors the intervention group and a negative number favors the comparison group. 
The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who are given the 
intervention (measured in standard deviations of the outcome measure). An indicator of the effect of the intervention, the improvement index can be interpreted as the expected 
change in percentile rank for an average comparison group student if that student had received the intervention. For example, an improvement index of +13 means that the 
expected percentile rank of the average comparison group student would increase by 13 points if the student received Bottom Line. A positive improvement index does not 
necessarily mean the estimated effect is statistically significant. Some statistics may not sum as expected due to rounding. 


@ Barr & Castleman (2017) did not require corrections for clustering nor difference-in-differences adjustments. The p-values for college enrollment and progressing in college 
were calculated by the WWC. This study is characterized as having a statistically significant positive effect on college enrollment and progressing in college because the 
estimated effects are positive and statistically significant. For more information, please refer to the WWC Procedures Handbook, version 4.0, page 22. 


In What Context Was Bottom Line Studied? 


The following section provides information on the setting of This information can help educators understand the context in 
the one study of Bottom Line that meets WWC standards, and a which the study of Bottom Line was conducted, and determine 
description of the participants in the research. whether the program might be suitable for their setting. 


WHERE THE STUDY WAS CONDUCTED 


1 study, 2,422 students in Boston, New York City, and Worcester, Massachusetts 


urban districts 


Race Ethnicity 


Black Asian Other Hispanic Non-Hispanic 


Gender Grades 
67 2 BPs) 


Female Male Grade 11—Postsecondary (PS) 


Details of Each Study that Meets WWC Standards 


This section presents details for the study of Bottom Line Findings from Barr & Castleman (2017) show evidence of 
that meets WWC standards. These details include the full a statistically significant positive effect of Bottom Line on 
study reference, findings description, findings summary, college enrollment (Table 5). This finding is based on an 

and description of study characteristics. A summary of outcome analysis that includes 2,422 students. The finding 
domain findings for the study is presented below, followed on progressing in college, which also shows evidence of a 
by a description of the study characteristics. These study- statistically significant positive effect, is based on an out- 
level details include contextual information about the study come analysis that includes 1,429 students. The findings and 
setting, methods, sample, intervention group, comparison research details summarized for this study come from three 
group, outcomes, and implementation details. For related citations, including the primary study listed above. 
additional information, readers should refer to the original See the References section, which begins on page 8, for a list 
study. of all related publications. 


Research details for Barr & Castleman (2017) 


Barr, A., & Castleman, B. (2017). The bottom line on col- 

lege counseling. Boston, MA: Bottom Line. Retrieved from 
https://www.bottomline.org/sites/default/files/The%20 
Bottom%20Line%200n%20College%20Counseling%20RCT- 
Paper 10 2017.pdf 


Table 5. Summary of findings from Barr & Castleman (2017) 


Meets WWC Group Design Standards Without Reservations 


Study findings 
; Sample Average Improvement Statistically 
Outcome domain size effect size index significant 
College enrollment 2,422 students 0.34 13 Yes 
Progressing in college 1,429 students 0.33 13 Yes 


Table 6. Description of study characteristics for Barr & Castleman (2017) 


WWC evidence 
rating 


Setting 


Methods 


Study sample 


Intervention 
group 


Comparison 
group 


Outcomes and 
measurement 


Additional 
implementation 
details 


Meets WWC Group Design Standards Without Reservations. This is a randomized controlled trial (RCT) with low 
attrition.® For more information on how the WWC assigns study ratings, please see the WWC Procedures and Standards 
Handbooks (version 4.0) and WWC Standards Briefs, available on the WWC website. 


The study was conducted with two cohorts of high school students, representing the graduating classes of 2015 and 2016 
in Boston, New York City, and Worcester, Massachusetts. Students who received the intervention were encouraged to 
attend a Bottom Line target college where they could continue the program; approximately 50 percent of students in the 
intervention group did so. There are about 30 of these target colleges, and they were described in the study as ones that 
offered an optimal combination of quality and affordability. These institutions are located in the same geographic region 
as the study high schools; examples include Boston University, the State University of New York at Albany, and target 
campuses in the City University of New York and University of Massachusetts systems. 


The study is a student-level RCT conducted across multiple high schools and colleges implementing the Bottom Line 
program. The sample frame included students who submitted applications to participate in Bottom Line prior to the start of 
their senior year, during one of three time periods that closed in May 2014, August 2014, or August 2015. The majority of 
students learned about Bottom Line through word of mouth referrals, but the program also engaged in active recruiting at 
local college fairs and at selected schools, and advertised the program via signage in local communities and through radio 
commercials. Eligible students were then randomly assigned to receive an invitation to receive Bottom Line services or not. 
Randomization was conducted with two groups: students who graduated high school in 2015 (cohort 1) and students who 
graduated in 2016 (cohort 2). The randomization procedure took into account the minimum number of students expected to 
be served in each of the Bottom Line cities (Boston, New York City, and Worcester, MA), resulting in differing probabilities of 
assignment across the three cities. 


The total sample included 2,422 students; 1,687 of these students were assigned to the Bottom Line program and the 
remaining 735 students were assigned to the comparison group. Cohort 1 included 995 students assigned to the Bottom 
Line program and 434 students assigned to the comparison group. Cohort 2 included 692 students assigned to the Bottom 
Line program and 301 students assigned to the comparison group. 


The high school students in the sample were from families that earn less than 200 percent of the federal poverty guidelines. 
Program eligibility required that students have earned at least a 2.5 high school GPA. About 81 percent of students in the 
sample were first-generation college students; approximately 70 percent were female and 32 percent Hispanic. Thirty-two 
percent of students in the sample were Black, 24 percent were Asian, and race was not specified for 44 percent of students. 


The Bottom Line college advising model provides two programs: Bottom Line Access for high school juniors and seniors 
and Bottom Line Success for students who attend a target college. The study followed students who were randomly 
assigned to the Bottom Line Access program as they transitioned to the Bottom Line Success program. Advisors interacted 
with students during high school, on average, 13 times during a 15-month period in which the intervention was delivered, 
starting with May of the student's junior year of high school through August of the year of graduation. Most meetings 
involved working on college applications or financial aid. Overall, this entailed 10-15 hours of contact time between advisors 
and students. After high school graduation, students who chose to attend a Bottom Line target college were matched to a 
new advisor at their college to continue to receive advising through Bottom Line Success. 


Students in the comparison group had access to existing college advising support services. The study surveyed Cohort 1 
students and found that almost all respondents in both groups applied for college (100 percent of intervention students and 
99 percent of comparison students) and financial aid (99 percent of intervention students and 97 percent of comparison 
students). The authors did not administer the survey to students in Cohort 2. 


Study authors reported findings on two outcome measures that are eligible for review under the Transition to College topic 
area. These outcomes include enrolled in a 4-year college (college enrollment domain) and continuously enrolled for the 
three semesters following high school (progressing in college domain). 


The study also reported supplemental findings including enrolled in any college, enrolled in a 2-year college (college 
enrollment domain), enrolled in any college for a second year, enrolled in a 2-year college for a second year, and enrolled 

in a 4-year college for a second year (progressing in college domain). A summary of these findings is available on the WWC 
website (httos:/whatworks.ed.gov). The supplemental findings do not factor into the intervention’s rating of effectiveness. 


Three study-reported findings, applied to college, filled out the Free Application for Federal Student Aid (FAFSA), and 
number of college applications completed (college readiness domain), did not meet WWC group design standards because 
analytic sample sizes were not reported. 


Bottom Line is a 501(c)(3) privately funded organization that serves low-income and first-generation students. The 
organization currently serves students in Boston, New York City, Chicago, and Worcester, Massachusetts. 
httos://www.bottomline.org/what-we-do. Financial support for the study was provided by the Michael & Susan Dell 
Foundation, the Coalition for Evidence-Based Policy, and the Laura and John Arnold Foundation. 
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Endnotes 


‘McFarland, J., Hussar, B., Zhang, J., Wang, X., Wang, K., 
Hein, S., Diliberti, M., Forrest Cataldi, E., Bullock Mann, 
F., and Barmer, A. (2019). The Condition of Education 
2019 (NCES 2019-144). U.S. Department of Education. 
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Retrieved from 
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? The descriptive information for this intervention 
comes from Barr & Castleman (2017). The What Works 
Clearinghouse (WWC) requests developers review the 
intervention description sections for accuracy from their 
perspective. The WWC provided the developer with the 
intervention description in August 2020 and the WWC 
incorporated feedback from the developer. Further 
verification of the accuracy of the descriptive information 
for this intervention is beyond the scope of this review. 


3 The literature search reflects documents publicly available 
by May 2020. Reviews of the studies in this report used 
the standards from the WWC Procedures and Standards 
Handbook (version 4.0) and the Transition to College 
review protocol (version 4.0). The evidence presented 
in this report is based on available research. Findings 
and conclusions could change as new research becomes 
available. 


4The target colleges for Worcester, Massachusetts are the 
same as those for Boston. 


5 The effects of Bottom Line are not known for other outcome 
domains within the Transition to College topic area, 
including middle school academic achievement, high 
school academic achievement, middle school attendance, 
high school attendance, staying in high school, progressing 
in high school, completing high school, college attendance, 
college academic achievement, college degree attainment, 
and labor market outcomes. 


6 The study is a randomized controlled trial with no attrition. 
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